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Abstract 

This study investigated interactive effects of navigation and offline comprehension skill on digital 
reading performance. As indicators of navigation, relevant page selection and irrelevant page 
selection were considered. In 533 Spanish high school students aged rr-17 positive effects of 
offline comprehension skill and relevant page selection on digital reading performance were 
found, while irrelevant page selection had a negative effect. In addition, an interaction between 
relevant page selection and offline comprehension skill was found. While the effect of relevant 
page selection was strong in good offline comprehenders, it was significantly reduced in weak 
offline comprehenders. The effect of offline comprehension skill was strong in students showing 
high rates of relevant page selection, while it was weak and insignificant in students showing low 
rates of relevant page selection. 
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Introduction 

In learning scenarios where materials are delivered online, e.g. in distance education 
programmes, students frequently need to progress on their task by navigating through a network 
of linked materials (or nodes). Specifically, during navigation they have to assess the relevance of 
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the available hyperlinks, and decide which one may contain information useful to answer the 
question. If students access a node with relevant information, they may still have to assess if the 
information obtained from this node is enough, or if they need to access further relevant links to 
provide a complete answer. Think of a person who considers taking up studies in, e.g., Psychology 
in a distance learning setting, and accesses the website of, e.g., the Open University. Even before 
enrolling in a study program, they will need to access a number of hyperlinks, and read and 
integrate the documents opened by these, to find out about course contents, fees, terms of 
application, etc. After having enrolled, they will need to complete study assignments sent to them 
online. Again, completing these assignments, e.g., when searching for literature to complete an 
essay, they will need to engage with navigating documents online by accessing certain hyperlinks, 
and discarding others when they are not relevant to the task at hand. Thus, theoretically, 
navigation is a crucial process in online and distance learning. In addition, traditional or offline 
comprehension skills are needed to process the documents accessed through the navigation 
process (e.g., Salmeron & Garcia, 2011). This means that in online learning scenarios as well, 
students need to decode words, parse the syntax of sentences, and execute local and global 
coherence processes to finally understand a document’s contents (e.g., Kintsch, 1998). This article 
aims at a contribution to clarifying how navigation processes and offline comprehension skills 
interact in online learning scenarios. In the following section, we first review the existing 
literature on this interaction. 

Evidence for the Interaction of Offline Comprehension Skills and Navigation 
Processes 

There is clear evidence that students’ navigation, as indicated by their traces in log files, play a 
major role in online question-answering tasks (Organization of Economic Co-Operation and 
Development [OECD], 2011). Specifically, a large scale study involving the adolescents 
participating in the OECD Programme for International Student Assessment (PISA) 1 2009 
electronic reading assessment revealed that students who displayed a more task-oriented 
navigation behaviour, as indicated by more visits to task-relevant pages, correctly responded to a 
higher number of questions. This result is in line with several other small scale studies that have 
looked at how adolescents and undergraduate students navigate in different learning tasks. These 
tasks include searching for information (Cress & Knabel, 2003; Hsu & Schwen, 2003), studying 
for a semester-long course (Puntambekar & Goldstein, 2007), reading for comprehension 
(Salmeron, Canas, Kintsch, & Fajardo, 2005; Salmeron, Kintsch, & Canas, 2006), reading to 


PISA is an international assessment of student skills that is carried out every three years since the 
year 2000 in all OECD and a number of partner countries and economies. The core assessment is in 
mathematics, science, and reading in students aged 15 , which in most countries means the end of 
compulsory education. See OECD ( 2010 , 2011 ) for details on PISA 2009 . 
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prepare a summary (Richter, Naumann, & Noller, 2003; Naumann, Richter, Flender, 
Christmann, & Groeben, 2007), or performing a science inquiry task (Goldman, Braasch, Wiley, 
Graesser, & Brodowinksa, 2012). In addition, OECD (2011) showed that students with good 
offline comprehension skills, as measured by a paper and pencil reading comprehension test, 
employed better navigation paths, which in turn improved even more their online performance. 
While this mediating role of navigation on the relationship between offline comprehension skills 
and online performance has been already reported in small scale studies (Naumann, Richter, 
Christmann, & Groeben, 2008; Salmeron & Garcia, 2011), less is known about the potential 
moderator effect of offline comprehension skills on the relationship between navigation and 
performance (e.g., navigation may predict online performance only for students with sufficient 
offline comprehension skills). The goal of this study is to shed light on this aspect. 

The fact that navigation, as measured by log files, predicts students’ performance in several online 
learning tasks has raised the attention of scholars due to its important implications on a 
theoretical and on a practical level (e.g. Giiyer, Atasoy & Somyiirek, 2015). On a theoretical level, 
navigation is important because it is highly interlinked with self-regulation learning (SRL) in 
online scenarios. SRL models describe students’ decisions in complex learning scenarios, such as 
deciding which hypertext sections to study and which one not. These decisions are based on 
students’ reflections on the constraints imposed by the scenario, their task goals, and the revision 
of their performance during the task (Winne & Hadwin, 1998). In addition, SRL models assume 
that learners try to optimize both achievement and how achievement is reached (Winne & Baker, 
2013). In online learning, readers are provided with tasks that require a self-directed selection 
and organization of text materials. In terms of achievement, there is ample evidence that 
increased self-regulation in online learning goes together with better learning outcomes (e.g. 
Azevedo & Cromley, 2004; Azevedo, Greene, & Moos, 2007; Azevedo, Moos, Greene, Winters, & 
Cromley, 2008). In terms of how achievement is reached, there is evidence that better self¬ 
regulation goes together with better navigation, and that better navigation partly mediates effects 
of self-regulation on learning outcomes in online learning scenarios (Richter, Naumann, Brunner 
& Christmann, 2005; Salmeron, Kintsch, & Kintsch, 2010). Thus, altogether, the analysis of 
navigation behavior can provide rich information to self-regulated models of learning (Winne, 
2010). 

On a practical level, the fact that navigation, as measured by log files, predicts students’ 
performance in several online learning tasks makes it useful for intelligent online systems. Log 
files can be traced to build user models, and recommendations can be provided during the 
learning session on the basis of students’ navigation. For example, if the system detects that a 
student is navigating task-irrelevant nodes, it may react, prompting her to focus on more relevant 
ones (e.g. Puntambekar & Stylianou, 2005; Vidal-Abarca et al., 2014; Winne, & Hadwin, 2013). 

The current study questions the widespread assumption that navigation univocally predicts 
performance in question-answering online tasks. Specifically, we tested the assumption that the 
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positive relationship between navigation and performance may be moderated by students’ offline 
comprehension skills. To the best of our knowledge, the potential moderator effect of 
comprehension skill on the relation between navigation and performance has not yet been 
investigated. Theoretically, moderation takes place when offline comprehension skills and 
navigation interact statistically in predicting online reading performance. A student might lack 
offline comprehension skills, but still have the skill necessary to navigate a digital text. For 
example students may match words in the question to words in the text, without a thorough 
examination of the relevance of that particular text section (Cerdan, Gilabert, & Vidal-Abarca, 
2011; Salmeron, Cerdan, & Naumann, 2015), or they may use typographical cues of hyperlinks to 
guide their navigation (Rouet et al., 2011). In this case, however, despite an apparently good 
navigation, comprehension outcomes will be poor. This is because a text that is ‘created’ through 
a readers’ self-guided selection and ordering of text materials, i.e. navigation, has to be read and 
understood eventually. Thus, according to this reasoning the effect of navigation on digital 
reading performance will be conditional on offline comprehension skills. In a student with good 
offline comprehension skills, a navigational path containing a large percentage of task-relevant 
materials will lead to good comprehension outcomes. A student with poor offline comprehension 
skills, in contrast, will not be able to utilize such a text. 

Our study aimed at testing the hypothesis that the relation between navigation and performance 
is moderated by students’ offline comprehension skills. To do so we used a large (IV=533) age- 
stratified sample comprising students aged eleven to 17 (see Table 1). 


Method 


Subjects 

533 Spanish students participated in the study. Students were sampled from six different schools 
in the province of Valencia, Spain. Within each school, one complete classroom of 7 th , 8 th , 9 th , and 
10 th graders was sampled. Schools and children were recruited to participate under specific 
agreement of collaboration between principals, regional educational authorities and the research 
team. 

Table 1 

Descriptive statistics and correlations of all variables in the study. 


Correlations 


M S Min Max l. 2. 3. 4. 5. 6. 7. 8. 
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1. Computer 

used” 

1.00 

0.04 

0.00 

1.00 








2. Computer 

available 8 

0.99 

0.09 

0.00 

1.00 

50*** 







3. Internet 

available 8 

0.97 

0.17 

0.00 

1.00 

.25*** 

12** 






4. Gender b 

Predictor 

and control 

variables 

0.46 

0.50 

0.00 

1.00 

-•05 

-.01 

-03 





5. Relevant 

page selection 

0.55 

0.15 

0.05 

0.89 

.01 

•05 

.02 

•13** 




6. Irrelevant 

page selection 

0.11 

0.07 

0.00 

0.33 

.01 

•05 

.01 

.16*** 

.07 



7. Offline 

comprehen¬ 
sion skill 

12.15 

4.38 

1.00 

20.00 

-.01 

.02 

.01 

.01 

•49 

*** 

-.18 

*** 


8. Age 

13.61 

1.26 

11.00 

17.00 

.06 

.01 

-■03 

-.01 

.21 

-.16 

•32 










*** 

*** 

*** 

Criterion 

9. Digital 

reading 

performance 

0.52 

0.20 

0.08 

1.00 

.00 

•03 

.08 

.08 

•63 

*** 

-•15 

*** 

•55 

*** 


Note. N = 533. Relevant page selection: Percentage of task-relevant pages visited. Irrelevant page 
selection: Percentage of task-irrelevant pages visited. Digital reading performance: Percentage of 
correctly answered questions in the digital reading test. 

a o = No, 1 = Yes. b o = female, 1 = male. 

** p < .01, *** p < .001 (two-tailed). 

As can be seen from Table 1, students’ age varied between eleven and 17 years (Table 1, row 8). 
Roughly half of the students were each male and female (Table 1, row 4). Most students had 
access to computers and the Internet in their homes (Table 1, rows 2-3). Also, most students had 
used a computer before (Table 1, row 1). 

Materials and Measures 
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Digital reading performance. Digital reading performance was measured through a series 
of tasks from a test of digital reading skills developed for this project. The current study forms 
part of a project aimed at developing a test of digital reading skills for secondary school students. 
The test consists of 36 items covering the aspects identified in the PISA framework (OECD, 
2009): accessing and retrieving (12 items), interpreting and integrating (13 items), and reflection 
and evaluation of information (11 items). Items are distributed into four scenarios that 
emphasized different uses of digital reading: educational (Wikipedia), multiple documents 
(Google), social (web forums) and navigation (web portal). The materials used in the test 
underwent a strict procedure of construction and validation (Salmeron et al., 2012): they were 
developed and subsequently tested in three pilot studies involving 60-80 students in each case. A 
final version of the test was tested in a sample of 555 students. As a measure of concurrent 
validity, we first correlated the scores students’ performance with the scores obtained with a 
standardized reading comprehension test (Llorens et al., 2011). Pearson correlation was .65, p < 
001. Additionally, we correlated students’ performance and students’ grades on Spanish (.38), 
Social sciences (.39), Natural sciences (.39), and Mathematics (.31) (all correlations p < .01). 
Results from that sample indicated that question difficulty ranged from .18 to .89 (i.e. success 
rate). Finally, the reliability of the questions (Cronbach ) was .78. 

In this study we used the web portal scenario of the test. Specifically, the materials consisted of a 
total of 12 items that students solved navigating through a web portal for youngsters. The web 
portal was organized around five main sections (Environment, Technology, Health, Sports, 
Courses), each of them including three subsections, which were further divided in three 
additional content pages. Some of the content pages included links to pages located outside the 
web portal. Thus, students navigated through a variety of web pages, including news pages, 
commercial sites and Google Maps mocks. Items assessed accessing and retrieving (5 items), 
integrating and interpreting (4 items), and reflecting and evaluating textual information (3 
items). For example, one accessing and retrieving item requested: ‘The Youth Web has just 
released a report about endangered birds and mammals. What is the level of endangered species 
in Canada?’ To answer the question, students had to access the following route starting from the 
home page of the Youth Portal: Environment > Endangered species > Threatened animals across 
the world, while avoiding other irrelevant links such as ‘Endangered animals: Iberian Lynx’. In 
terms of the PISA framework, all content pages were authored texts; 66% tasks related to 
continuous texts and 33% related to non-continuous texts. In addition, all tasks required between 
two and six navigational steps to find the target information (M = 2.92 steps, SD = 1.16), i.e., all 
tasks related to multiple web pages. All tasks were presented in a mimicked browser. Each item 
referred to information from a particular subsection that was not targeted by any other item 
before. The reliability of the items (Cronbach’s ) was .60. 

Offline comprehension skills. Offline comprehension skills was measured through the 
Reading Literacy test for Secondary Education (CompLEC) by Llorens et al. (2011). The test is 
composed of five units, three continuous and two non-continuous texts. Topics covered included 
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a textbook excerpt about global warming, an article from a popular science magazine about the 
language of the bees, a pamphlet about the postural ergonomics of work chairs, two letters to the 
editor about the pros and cons of nuclear energy, and a newspaper diagram about traffic 
accidents. Each unit included 3-5 questions covering the different aspects suggested by the PISA 
framework: accessing and retrieving (5 items), integrating and interpreting (10 items), and 
reflecting and evaluating textual information (5 items). The test has been validated with a sample 
of 1854 students from 42 different high-schools. Its reliability in the present sample was .80 
(Cronbach’s ). 

Navigation. Log files were used to analyze students’ navigation behavior. We used as an 
index of the navigation quality the percentage of task-relevant pages visited for each item 
(relevant page selection). The number of task-relevant pages visited reflects the process of task- 
accurate selection of learning or reading materials from online sources and can thus be assumed 
one crucial aspect of the navigation process, and has thus been shown to be associated with the 
quality of comprehension and learning (e.g. Cress & Knabel, 2003; Goldman et al., 2012; 
Naumann et al., 2008). Pages were considered relevant for a particular question if students 
needed to access them in order to locate information useful to answer that question. Thus, this 
index not only considers if a participant accessed or not the page or pages that contained useful 
information, but more precisely the extent to which she followed the path leading to those useful 
pages. To rule out the possibility that this index merely reflects the length of a navigational path, 
we considered in addition the percentage of irrelevant pages that were accessed by a student in 
each task (irrelevant page selection). While we predicted a positive effect of relevant page 
selection on digital reading performance, we predicted a negative effect of irrelevant page 
selection. 

Procedure 

Students were tested in their classrooms during school hours in two one-hour sessions. In a first 
session, students completed the offline comprehension skills test. After completion, they filled in 
two questionnaires asking for demographic information and about the frequency of their use of 
computers. In a second session, students completed the digital reading performance measure in 
the schools’ computer lab. Data was collected using USB devices. 

Statistical Analysis 

To test our hypothesis about the moderator role of offline comprehension skills on the relation 
between navigation and performance, we estimated a linear mixed model, using the R 
environment (R Development Core Team, 2012) with the package lme4 (Bates, Maechler, & 
Bolker, 2012). The predictor variables offline comprehension skill, relevant page selection, and 
irrelevant page selection were entered as fixed effects. In addition, the interactions of offline 
comprehension skill with relevant page selection and irrelevant page selection were entered as 
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fixed effects. Since the predictor variable offline comprehension skill could be expected to covary 
with age, and thus be confounded with other age-related variables, we entered students’ age as a 
control variable. All predictor and control variables were z-standardized. Since complete 
classrooms were sampled, we entered a random classroom intercept to account for dependencies 
between students coming from the same classroom. As a measure of effect size, we report as a 
default the proportion of variance uniquely accounted for by each fixed effect (marginal R 2 
following Nakagawa and Schielzeth, 2013). The significance of the random classroom intercept 
effect was evaluated using a simulation-based exact likelihood test with the package RLRsim 
(Scheipl, Greven, & Kuechenhoff, 2008). 


Results 

Descriptive statistics and correlations for all variables in the model are given in Table 1 (rows 5- 
9 )- 


Recall that our hypothesis claims that the effect of relevant page selection on online 
comprehension is enhanced by offline comprehension skills, since without those appropriate 
skills, students displaying whatever navigation behavior will not be able to completely understand 
a digital text. This means that in a regression model with offline comprehension skills and 
relevant page selection as predictors, plus an interaction term, a positive regression coefficient 
should emerge for the interaction, indicating an increasing impact of relevant page selection on 
digital reading performance with increasing offline comprehension skills. In contrast, no such 
interaction effect was expected for irrelevant page selection. Selection of irrelevant pages can be 
seen as detrimental to digital reading performance, regardless of a student’s offline 
comprehension skill. Model results are summarized in Table 2. 

Table 2 

Model results 


Fixed effects 

Effect 

(SE) 

z 

R 2 m 

Intercept 

-0.48 

(0.01) 



Relevant page selection 

0.13 

(0.01) 

14.12*** 

0.22 

Irrelevant page selection 

-0.02 

(0.01) 

-3.40** 

0.01 

Offline comprehension skill 

0.05 

(0.01) 

6.70*** 

0.05 

Offline comprehension skill x Relevant page 




0.02 

selection 

0.03 

(0.01) 

4.11*** 


Offline comprehension skill x Irrelevant 

-0.01 

(0.01) 

-0.82 
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page selection 



Age 

0.01 (0.01) 

1.40 

Random effects 



Classroom random intercept 

0.001* 


Model fit 

R 2 m = .51, R\ = -53 



Note. R 2 m : Marginal R 2 , variance explained by fixed effects. R 2 C : Conditional R 2 , variance 
explained by fixed and random effects (Nakawa & Schielzeth, 2013). Effects sizes (decrements in 
variance explained by fixed effects) are reported for significant effects only. 


* p < .05, ** p < .01, *** p < .001. 

The random effect for classroom was significant, meaning that there was systematic between- 
classroom variance in digital reading performance over and above what was explained by the 
fixed effects in the model (see Table 2). 

Analysis of fixed effects revealed a large positive effect of relevant page selection (Table 2, row 3), 
while irrelevant page selection had a negative effect as expected (Table 2, row 4), which was 
however small. Also as expected, offline comprehension skill had a medium-sized positive effect 
(Table 2, row 5). In addition to these main effects, there was a positive effect for the interaction 
between relevant page selection and offline comprehension skill (Table 2, row 6). The interaction 
between offline comprehension skills and irrelevant page selection was not significant (Table 2, 
row 7). The control variable age had no effect over and above the other predictors in the model 
(Table 2, row 8). 

In line with our hypothesis, the positive sign of the regression coefficient for the interaction 
between relevant page selection and offline comprehension skills indicated that positive effects of 
relevant page selection on digital reading performance were accelerated with increasing offline 
comprehension skills. Accordingly, positive effects of offline comprehension skills were 
accelerated with increasing rates of relevant page selection. To further interpret this interaction, 
we conducted simple slopes analyses (see Aiken & West, 1991). 

Firstly, we estimated the effects of relevant page selection on digital reading performance 
conditionally on high offline comprehension skills (one standard deviation above the mean, see 
the straight line in Figure la) and low offline comprehension skills (one standard deviation below 
the mean, see the dotted line in Figure la). 
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relevant page selection (z-standardired) offline comprehension skil (z-standardized) 

Figure l : (A) Simple slopes for the moderated regression of digital reading performance on 
relevant page selection, conditional on high offline comprehension skill (+ 1 SD, straight line), 
and low offline comprehension skill (-1 SD, dotted line). (B) Simple slopes for the moderated 
regression of digital reading performance on offline comprehension skill, conditional on high 
rates of relevant page selection (+ l SD, straight line), and low rates of relevant page selection (-1 
SD, dotted line). See text for significance tests. 

For students high in offline comprehension skills, a strong effect of relevant page selection on 
digital reading performance was found, |3 = 0.17 (SE = 0.01), t = 12.50, p < .001. For students low 
in offline comprehension skills, this effect was still significant, but reduced to about 60% of its 
size, p = 0.10 ( SE = 0.01), t = 9.44, p < .001. 

Secondly, we estimated the effects of offline comprehension skill on digital reading performance 
conditionally on high rates of relevant page selection (one standard deviation above the mean; see 
the straight line in Figure lb), and low rates of relevant page selection (see the dotted line in 
Figure lb). For students selecting many relevant pages (one standard deviation above the mean), 
there was a strong effect of offline comprehension skill on digital reading performance, |3 = 0.08 
(SE = 0.01), t = 8.00, p < .001. For students selecting few relevant pages, in contrast, the effect of 
offline comprehension skill was not significant, |3 = 0.02 (SE = 0.01), t = 1.73, p > .05. 


Discussion 

To the best of our knowledge, the present study is the first to provide clear evidence that the 
relationship between relevant page selection and performance in question-answering online tasks 
is moderated by students’ offline comprehension skills. While the results replicate the well- 
established relationship between navigation and performance in online tasks (see Naumann, 
2008, for a review), they reveal that the influence of relevant page selection can drop to 
approximately 60% if students have low offline comprehension skills. They do also reveal that 
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good offline comprehension skills are not sufficient in themselves to produce good digital reading 
performance. Rather, if students fail to comply with demands of relevant page selection, the 
otherwise strong and positive association of offline comprehension skill and digital reading 
performance is no longer significant. 

One possible explanation for these results is a threshold model. According to such a model, when 
a critical threshold of offline comprehension skills is reached, students can benefit from their 
efficient navigation to improve their digital reading performance. It is a reasonable assumption 
that students would need to score on the PISA reading proficiency level 3 (out of 6) or above. This 
means, for instance, that they are able to, e.g., find multiple pieces of information in a text, or link 
information from different texts to one another (see OECD, 2010, for a detailed description). This 
assumption however needs to be corroborated by future research as the offline comprehension 
skills measure used in this study is not linked to the PISA reading scale. 

It is strong comprehenders who benefit from efficient navigation of task relevant pages. Why do, 
in contrast, poor readers not benefit so much from their navigation? From the perspective of a 
developmental model, one can argue that these students still fail on matching their navigation 
goals to the available links by means of inferences. Thus, their assessments of a link’s relevance 
might rely on more superficial processes, such as links’ typography (Rouet et al., 2011) or word 
matching (Cerdan et al., 2011; Salmeron et al., 2015), e.g., if the goal and the hyperlink share a 
particular word. By using those cues to navigate, students may still access pages that are relevant 
for their goal, but nonetheless they won’t be able to make complete sense of the information on 
those pages. Log files provide information about the percentage of relevant pages visited by 
students, but they don’t give a complete picture of which cognitive operations students actually 
use to select those pages. A possible way to clarify this issue is to combine the information from 
log files with other online measures, such as students’ verbal protocols (e.g. van Gog, Paas, van 
Marrienboer, & Witte, 2005; Richter et al., 2005). Future research should clarify this issue. 

In claiming a moderation effect of offline comprehension skills for the relationship between 
navigation and online performance, some precautionary remarks are in place as well. First, while 
it was one crucial property of navigation that was investigated in the study reported here, the 
selection of relevant (and irrelevant) text materials, this is by no means the only important 
property of a navigational path. For instance, coherence of the navigational path has been 
established as another variable that is predictive of comprehension and learning outcomes in 
reading and comprehending digital text (Amadieu & Tricot, 2008; Naumann et al., 2007; Richter 
et al., 2005; Salmeron et al., 2005; Salmeron, Kintsch, & Canas, 2006), as has the use of 
navigation help devices such as topical overviews (Dias & Sousa, 1997; Salmeron & Garcia, 2011; 
Su & Klein, 2006). For these properties of navigation, for the time being, we do not know 
whether, and how, their impact on performance is moderated by students’ skills. 
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Second, although in our study offline comprehension skills moderate the relationship between 
navigation and performance, navigation has a positive effect in most students. Other patterns of 
moderation can be observed. For example, a particular navigation pattern may be useful for 
students with or without a particular skill, but may have no effect on others. Although it is not 
directly linked to navigation, related evidence on the effect of overviews in hypertext exemplify 
these potential moderator patterns. In this line, Salmeron, Baccino, Canas, Madrid, & Fajardo 
(2009, study 1) found that providing a thematic overview for a hypertext had a positive effect on 
students’ comprehension, only if the hypertext was difficult and if students had low background 
knowledge on the topic. Finally, while offline comprehension skills is a strong predictor of 
navigation, research has identified other students’ individual variables that are also important to 
articulate an efficient navigation, and thus their potential role as moderators should be studied in 
the future. One such important variable is background topic knowledge (e.g., Coiro, 2011). We 
could expect that, given a similar percentage of relevant pages visited, students with high 
background knowledge may fill in the existing gaps in their response, while students with low 
background knowledge may have difficulties overcoming an incomplete response generated 
through their navigation (cf., Taub, Azevedo, Bouchet, & Khosravifar, 2014). 

Implications for Research on Navigation 

What are the potential implications for the moderated relationship between navigation and online 
performance? Here, the present research carries two messages. First, at a theoretical level, our 
results imply that similar navigation traces may be reflecting different cognitive and 
metacognitive processes. This is a challenge for using navigation traces to inform and enrich self¬ 
regulation models of learning. A way to limit the number of potential processes underlying the 
selection of a particular navigation path is to include theoretically motivated hyperlinks, the 
selection of which can be easily linked to a unique theoretical explanation (Winne, 2010). For 
example, a recent question-answering study using Wikipedia (Salmeron et al., 2015) included 
three types of hyperlinks embedded in relevant sections of the hypertext: a) relevant uncued links 
that paraphrased the main idea of the question, and that led to a hypertext node with useful 
information to answer the question; b) irrelevant cued links that included a word that matched 
with a word in the question, although it was not representative of the main idea of the question, 
and the link led to an irrelevant node, and c) irrelevant uncued links that didn’t include a 
paraphrase or a word match to the question. Results from the navigation patterns followed by the 
students allowed to identify which relevance cues (semantic and/or word matching) they were 
using to navigate the hypertext. 

Second, at a practical level, the existence of a moderation effect indicates that navigation alone, at 
least the navigation index used in this study, i.e., percentage of relevant pages accessed, can’t be 
used to accurately predict students’ performance in online question-answering tasks. An 
intelligent system that reacts to students’ navigation behavior, without considering important 
student variables, such as offline comprehension skills, will generate an inaccurate representation 
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of the user, which may lead to less useful recommendations. For example, if a system identifies 
that a user is deviating from the optimal path to find relevant information, recommending an 
optimal route may be useful if the student has the necessary skills to realize that she is in an 
unfruitful path, but it may be less efficient if the student does not have the necessary reading 
skills to do so. Thus, in designing distance learning materials clearly usability in terms of 
navigation is important, i.e., supporting students to actually identify the relevant parts of the 
materials given the aims of study. What might however be equally important as suggested by the 
present results is to look at the text’s readability and difficulty itself. Since online learners’ offline 
comprehension skills might be measured before students engage with the materials, they cannot 
be changed in the short term. Thus, facing students with comprehension difficulties, for a course 
designer, it will be of importance to create materials as easy to comprehend as possible. 
Otherwise, as suggested by the present results, increasing the materials’ usability in terms of 
navigation won’t help much. 

As we see it, the present analyses however already make a strong argument that while navigation 
does play a crucial role in digital reading performance, its effect is moderated by the more 
traditional offline comprehension skills. This also means that instructional programs which aim 
at strengthening students’ navigation processes, preparing them for a digital 21 st century world 
where open and distance learning scenarios are ubiquitous, should take students’ comprehension 
skills into account. Probably, students who struggle with comprehension will need support to 
strengthen their comprehension skills first, before they can benefit from instruction targeted at 
their preparedness to adequately select hyperlinks in online reading scenarios. 

Limitations 

Finally, some cautionary remarks about limitations of the present results are in place. First of all, 
our data base was correlational. This means that the present results should be backed, e.g., by 
training studies, where in the best case offline comprehension skill and navigation processes are 
manipulated experimentally. Second, while we analyzed data from high school students in this 
research, open and distance learning is frequently, and in the future will be, aimed at adults (e.g., 
Munoz, Redecker, Vourikari, & Punie, 2014). While there appears to be some evidence that in 
adults comprehension skills interact with navigation support (Naumann et al., 2007), little is 
known though about possible interactions between comprehension skill in navigation behavior in 
adult distance learners. Cleary, future research will have to shed light on this issue. 
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